This paper is in the following e-collection/theme issue:

Mobile Apps for Cancer Care and Cancer Prevention and Screening (196) Human Factors and Usability Case Studies (731) Usability of Apps and User Perceptions of mHealth (884) mHealth in a Clinical Setting (571) Usability Evaluation Case Studies (180) Patient Education for Cancer (137)

Published on in Vol 4, No 2 (2018): Jul-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/10932, first published .
Usability Evaluation of a Mobile Phone–Based System for Remote Monitoring and Management of Chemotherapy-Related Side Effects in Cancer Patients: Mixed-Methods Study

Usability Evaluation of a Mobile Phone–Based System for Remote Monitoring and Management of Chemotherapy-Related Side Effects in Cancer Patients: Mixed-Methods Study

Usability Evaluation of a Mobile Phone–Based System for Remote Monitoring and Management of Chemotherapy-Related Side Effects in Cancer Patients: Mixed-Methods Study

Authors of this article:

Saeed Moradian1, 2 Author Orcid Image ;   Monika K. Krzyzanowska3, 4 Author Orcid Image ;   Roma Maguire5 Author Orcid Image ;   Plinio P. Morita6, 7 Author Orcid Image ;   Vishal Kukreti8, 9 Author Orcid Image ;   Jonathan Avery8 Author Orcid Image ;   Geoffrey Liu8, 10 Author Orcid Image ;   Joseph Cafazzo11, 12 Author Orcid Image ;   Doris Howell2, 3 Author Orcid Image
Article Authors Cited by (36) Tweetations (2) Metrics

    Original Paper

    1Department of Supportive Care-Research Division, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

    2The Lawrence S Bloomberg Faculty of Nursing, University of Toronto, Toronto, ON, Canada

    3Department of Medical Oncology and Hematology, Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

    4Institute of Medical Science, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

    5Department of Computer and Information Sciences, University of Strathclyde, Glasgow, United Kingdom

    6School of Public Health and Health Systems, University of Waterloo, Waterloo, ON, Canada

    7Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada

    8Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

    9Division of Medical Oncology and Hematology, Department of Medicine, University of Toronto, Toronto, ON, Canada

    10Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

    11Centre for Global eHealth Innovation, University Health Network, Toronto, ON, Canada

    12Institute of Health Policy, Management & Evaluation, University of Toronto, Toronto, ON, Canada

    Corresponding Author:

    Doris Howell, BS, RN, PhD

    Department of Medical Oncology and Hematology, Princess Margaret Cancer Centre

    University Health Network

    Toronto, ON,

    Canada

    Phone: 1 416 946 4501 ext 3419

    Fax:1 416 340 4739

    Email: Doris.Howell@uhn.ca


    Background: As most chemotherapy is administered in the outpatient setting, patients are required to manage related side effects at home without direct support from health professionals. The Advanced Symptom Management System (ASyMS) has been developed to facilitate the remote monitoring and management of chemotherapy-related toxicity in patients with cancer, using patient-reported outcomes questionnaires and a clinician alerting system.

    Objective: This study aims to evaluate the usability of the ASyMS, a mobile phone–based technology, from the perspective of Canadian patients with cancer receiving chemotherapy to identify existing design, functionality, and usability issues and elicit their views, experiences, and satisfaction with the ASyMS.

    Methods: We used a mixed-method approach to data collection with user-based testing, a think-aloud technique, semistructured interviews, and short answer questionnaires with a purposive sample of 10 patients with cancer. Participants attended usability testing sessions at the Centre for Global eHealth Innovation, University Health Network, and performed specific tasks on the ASyMS device. The test was videorecorded and each task was timed during the test. After the usability sessions, participants completed a posttest questionnaire and participated in a semistructured qualitative interview. A thematic analysis was used to code and categorize the identified issues into themes that summarized the type and frequency of occurrence.

    Results: The thematic analysis generated 3 overarching themes as follows: ASyMS user-friendliness; usefulness of ASyMS (content quality and richness); and intention to use. Results from the posttest questionnaire indicated that 80% (8/10) of participants had great motivation to use the ASyMS, 70% (7/10) had positive perceptions of the successful use of the ASyMS, and all (10/10, 100%) had a positive attitude toward using the ASyMS in the future. Most identified design and functionality issues were related to the navigation of the ASyMS device and a desire for a more attractive design with advanced functionality and features. The main general design recommendations were as follows: enhance the readability of the screen; implement advance options (eg, search option); and support better navigation.

    Conclusions: The ASyMS has shown positive perceptions of patients in usability testing and qualitative interviews. An evaluation of the effects of the ASyMS on symptom outcomes in a clinical trial is needed.

    JMIR Cancer 2018;4(2):e10932

    doi:10.2196/10932

    Keywords

    mobile apps; mobile health; mobile phone; patient-centered care; patient remote monitoring; self-care; symptom management; usability testing; mobile phone 


    Background

    Systematic chemotherapy continues to be the main treatment modality for almost all major cancer types [1]. Chemotherapy is associated with a myriad of symptoms and adverse treatment side effects that can range from mild to life-threatening, severe, and disabling [2]. Therefore, early recognition and effective management of these symptoms by both clinicians and patients are critical to reducing physical and psychological treatment sequalae [2].

    There is growing evidence in support of using patient-reported outcome measures (PROMs) for improving symptom management [3-5]. The increased number of mobile phone users creates opportunities for developing models of supportive care that use these technologies for monitoring PROMs to improve home-based, proactive “real-time” symptom monitoring and management [6-8]. Several Web-enabled PROMs systems have been trialed in oncological settings [4,8-10] and shown to support patients in managing chemotherapy-related symptoms [3,11], improve symptom control [11,12], and enhance patient-clinician communication [4]. However, few of the current systems have been developed in line with best-practice guidelines in user-centered design nor have verified the system usability during the stages of system development [5,13], which could impact their use by patients and clinical integration in practice settings [14].

    The aim of this study, which is part of a larger project to enhance the provision of timely, high-quality, person-centered supportive care, is to evaluate the usability of a mobile phone–based technology, the Advanced Symptom Management System (ASyMS), from the perspective of Canadian patients with cancer (colorectal and lymphoma) receiving chemotherapy in a controlled usability testing environment. The secondary aim of our end-user testing is to explore users’ performance and satisfaction with the system interface and their perspectives and experience with the system and the content of ASyMS [9]. In addition, this study assesses the ASyMS against a set of human factors design guidelines and heuristics to increase the likelihood of discovering more design features and function issues that could impact user experience and willingness to use the system.

    Advanced Symptom Management System

    The ASyMS, one of the more advanced remote monitoring systems, is a mobile phone–based device designed to monitor and manage chemotherapy-related toxicity in the home setting. It enables real-time remote monitoring of cancer symptoms using PROMs [15]. The ASyMs uses innovative risk prediction modeling and decision-support tools that allow for timely, high-quality, person-centered supportive care for better treatment toxicity management [10,16].

    Patients using the ASyMS complete an e-symptom PROMs questionnaire to assess the occurrence, severity, and distress associated with each symptom. After completing the questionnaire, patients immediately receive evidence-based, self-care advice on the mobile phone based on the specific symptoms reported, which facilitates the self-management of symptoms. Leveraging evidence-based algorithms, symptoms reported through the device that meet a threshold criteria (ie, high level of severity) trigger alerts to cancer care clinicians, usually nurses, who on the receipt of an alert can view patients’ symptom reports on a secure webpage and contact patients directly at home by telephone, enabling the initiation of proactive clinical interventions (Multimedia Appendix 1) [17].

    The ASyMS was developed in the United Kingdom based on the extensive patient and clinician engagement, and its utility and acceptability have been tested in UK populations [18,19]. The effect of the ASyMS intervention on patient outcomes is uncertain and is being tested in a large multisite trial in European countries [16]. We undertook a study to test the usability of the ASyMS program to identify its potential for the uptake in a Canadian cancer population. The ASyMS program was installed on an Android mobile phone with a 5.00-inch touchscreen display with a resolution of 720 pixels by 1280 pixels. Figure 1 and Multimedia Appendix 2 show a preview and some features and functions of the ASyMS.

    ‎
    Figure 1. Patient handset screenshots. (Source: Docobo Ltd).
    View this figure

    Study Design

    We used a mixed-method approach (qualitative and quantitative data sources) to increase the depth of evaluation and support methodological triangulation to improve the reliability and validity of findings [20]. Mixed-methods also allow for a more comprehensive understanding of participants experience and enable the identification of specific usability issues [20,21]. A usability study evaluates how a specific process or product works for individuals and the extent to which a user can use a product to achieve specific goals (interaction between user and task in a defined environment) [22,23].

    Data collection combined user-based testing using a think-aloud technique, semistructured qualitative interviews based on a qualitative descriptive methodology [24], and short answer quantitative questionnaires; all these methods have been used widely for usability testing [25]. Specifically, think aloud is a user-related method for assessing usability where users are encouraged to verbalize their perceptions out loud as they interact with the system [26]. Participants’ experiences with the system evaluated through qualitative interviews and questionnaires can inform potential for future uptake [27].

    Participants and Setting

    Estimation of the sample size for a usability test depends on several variables, including types of test users available, the mission criticality of a system (any factor that is essential for system operation), and problem discovery rate (the number of usability issues that can be uncovered by users) [28,29]. Although, it has been shown that 80% of the usability problems can be detected with 4 or 5 participants in a usability testing [30], Faulkner [29] found that the minimum percentage of identified usability issues increased from 55% to 82%, and the mean percentage of issues increased from 85% to 95% when the number of participants was increased from 5 to 10. Thus, in this study, we aimed to recruit a minimum of 10 patients. We used a purposive sampling method to ensure maximal variation in end-user characteristics, specifically younger (age <50 years) and older (age >50 years) adult patients with diverse cancer types (colorectal or lymphoma), males and females, and those with and without experience in using mobile technology.

    The Institutional Review Board Approval was obtained from the University Health Network (UHN) to conduct the study prior to recruitment (#15-9432). Patients were recruited from ambulatory follow-up clinics at the Princess Margaret Cancer Center, a cancer research center affiliated with the University of Toronto as part of the UHN. The inclusion criteria were that patients received, at least, one cycle of chemotherapy for treatment of their cancer (colorectal or lymphoma), were aged >18, and able to participate in usability testing for “think aloud” in English. All participants gave informed written consent for participation in the study.

    Data Collection

    The main goal of user-centered methods is to involve real users, elicit their views and experiences of the intervention to identify usability issues [31,32]. To meet the aim of this study, usability sessions were videorecorded from multiple angles, and participants were encouraged to share their thoughts verbally as they progressed through a set of predefined tasks (think aloud) [26]. We aimed to elicit feedback and identify design, functionality, and usability issues. In addition, participant experiences, thoughts, feelings, and satisfaction with the ASyMS were assessed by an audiotaped, semistructured, face-to-face qualitative interview with participants and through completing a short questionnaire (modified Telehealth Acceptance Measure, TAM), immediately after usability testing sessions (Multimedia Appendix 3).

    The TAM questionnaire comprises 10 questions on a Likert scale ranging from 1 to 7; higher scores indicate greater motivation to use telehealth, more favorable perceptions of the successful use of telehealth, greater patients’ belief that significant others would like them to use telehealth, and more positive attitude toward using telehealth. The TAM questionnaire is designed to assess patients’ motivation to use telehealth and includes questions that are derived from the theory of planned behavior, a model that explains the factors that underpin people’s motivation to act [33]. We used this questionnaire to indicate participants’ overall motivation and readiness to use the ASyMS device, assess participants’ perceived behavioral control, subjective norms, attitudes toward the device, and the extent to which individuals perceive that significant others want them to use the device.

    Usability Testing Procedure

    Participants attended usability testing sessions at the Healthcare Human Factors labs at the Centre for Global eHealth Innovation, UHN. Each participant was advised that the aim was to test the ASyMS device and not participants. In addition, participants received written and verbal information regarding the testing procedure, and a brief introduction to the ASyMS before usability testing commenced.

    Before starting the session, participants completed a demographic questionnaire. Participants were given a case scenario and a simulated symptom experience they might have during one of their chemotherapy cycles. Participants were requested to follow the tasks provided to them on the ASyMS device, representing typical user goals. Throughout testing, each participant was requested to perform specific tasks that consisted of the following: completing the e-symptom questionnaire (PROMs) on the ASyMS device; finding information about side effects and self-care; filling out the anytime section of the symptom questionnaire; and finding a history of side effects (Multimedia Appendix 4). A trained moderator guided participants through the testing procedure but did not intervene or disrupt the thinking-aloud process. Furthermore, from the observation room, behind a one-way mirror, 2 observers watched the interaction, made notes about what was verbalized, and observed to inform the analysis, and ensured the entire session was recorded. Each task was timed during the test.

    After the usability sessions, participants completed the posttest questionnaire to assess their perceptions about the usability of the ASyMS (Multimedia Appendix 3). In addition, they participated in a face-to-face interview regarding the utility and acceptability of the ASyMS in managing chemotherapy symptoms, parts of the content or aspects of the system they liked or disliked, and the reason for their response. The complete testing procedure for all steps averaged approximately 2 hours (range 1.5-2 hours).

    Data Analysis

    The audio and video recordings from the usability and interview sessions were transcribed. The thematic analysis was used to identify all emerging issues and the relations between the themes [34,35]. The identified issues were coded and categorized according to the type and frequency of occurrence [35]. Data collection and analysis continued until no more patterns or themes were emerging from the data [36]. Two members of the research team reviewed the transcripts. Any discrepancies between reviewers were resolved through discussion or the involvement of a third reviewer, if necessary. All qualitative data were coded using NVivo 10 qualitative data analysis software. In addition, a set of variables related to the participants’ performance, including the number of errors each participant made, requests for help, the time taken to complete the task, participant feedback, observers and moderator’s notes, and reviewing the videos, were used to identify a list of usability issues. Descriptive statistics (means, medians, ranges, frequencies, or percentages) were used to summarize these data.


    Participant Characteristics

    Table 1 presents the characteristics of the study participants. Of 10 participants, 7 were male and 3 were female, with an average age of 68 (range 18-78) years. Most participants (n=8) had higher education (college or university). All participants had their own mobile phone, of which 70% (7/10) had a smartphone, whereas 30% (3/10) owned a regular cell phone (not a smartphone). In addition, 60% (6/10) of participants mentioned that they were comfortable or very comfortable using these devices; 80% (8/10) were comfortable using the internet.

    Quantitative Results

    Using the video analysis, the task completion times, the number of errors made by participants while completing tasks, and the number of times they asked for help are shown in Table 2. We followed the TAM developers instructions to score and interpret the TAM (Multimedia Appendix 3). Overall, 80% of participants (8/10) scored >4 on Q2, Q4, and Q5 (mean=5.8), indicating high motivation to use the ASyMS device. In addition, 70% of participants (7/10) scored >4 on Q3, Q6, and Q7 (mean=5.6), indicating they had positive perceptions of the successful use of the ASyMS, and all participants (n=10) scored >4 on Q8 and Q9 (mean=6.1), showing they believed that significant others would like them to use the ASyMS. Furthermore, all participants (n=10) scored ≥5 on Q10 (mean=6.3), suggesting a positive attitude toward using the ASyMS device in future (Table 3).

    Table 1. Participants’ characteristics (N=10).
    CharacteristicsValue
    Age, median (range)68 (18-78)
    Sex, n

    		Male7

    		Female3
    Education, n

    		High school2

    		University or college5

    		Postgraduate degree (eg, Doctor of Philosophy)3
    Own a phone, n

    		Smartphone7

    		Regular cell phone3
    Hours use a computer each week, n

    		Not at all1

    		1-2 h1

    		4-5 h1

    		>7 h7
    Comfortable using a smartphone, n

    		Not at all1

    		A little comfortable3

    		Comfortable4

    		Very comfortable2
    Comfortable using a computer, n

    		Not at allN/Aa

    		A little comfortable4

    		Comfortable3

    		Very comfortable3
    Comfortable using the internet, n

    		Not at allN/A

    		A little comfortable2

    		Comfortable4

    		Very comfortable4
    Cancer type, n

    		Gastrointestinal cancer3

    		Lymphoma7

    aN/A: not available.

    Table 2. Quantitative results (time and SD, errors, and requests for help).
    TaskMean task completion time (SD) in secondsFrequency of error, nerror (n)aFrequency of requests for help, nhelp (n)b
    Task 1: Complete e-symptom questionnaire846 (135)25 (8)47 (9)
    Task 2: Find information about side effects and self-care502 (250)52 (9)32 (8)
    Task 3: Filling out anytime questionnaire232(124)35 (10)37 (10)
    Task 4: Find history of side effects257 (70)34 (10)26 (9)

    anerror represents the number of times an error was made, and n represents the number of people who made the error.

    bnhelp represents the number of times a request for help was made, and n represents the number of people who made the request for help.

    Table 3. Telehealth Acceptance Measure: Mean scoring.
    Participants’ characteristics

    		Behavioral intention itemPerceived behavioral control itemSubjective norm itemAttitude item
    NoAgeSexComfortable using a smartphone
    168FemaleA little comfortable76.377
    278MaleNot at all5.74.355
    318MaleVery comfortable66.35.56
    468MaleComfortable7667
    577FemaleA little comfortable5.336.56.3
    659MaleA little comfortable3.7776.7
    775MaleComfortable5.3665
    870MaleComfortable7777
    934MaleVery comfortable3.7356.3
    1055FemaleComfortable7767

    Qualitative Results

    The thematic analysis of the interview transcripts and participants’ feedback generated 3 overarching themes and related subthemes: ASyMS user-friendliness, with subthemes of design, navigation, and ease of use of the ASyMS; usefulness of the ASyMS (content quality and richness), with subthemes of self-care advice and information on the ASyMS, and appropriateness of the ASyMS questions; and intention to use, with subthemes of acceptance and satisfaction with using the ASyMS in future.

    Advanced Symptom Management System User-Friendliness

    Both the quantitative and qualitative data from the usability testing identified several design and functionality issues for the ASyMS’s device that may negatively impact its efficient use. Each of the recognized issues was mapped to source events (ie, participants’ feedback, errors, and moderator observation). Moreover, each of the issues was classified in one of the 8 usability heuristics for mobile devices (ie, match between system and the real world, ease of input, and screen readability) [37].

    The identified issues shown in Table 4 mostly relate to the navigation of the ASyMS device.

    Table 4. Identified usability issues.
    Problem

    		Category (usability heuristics)Source
    Introduction screen not intuitive nor informative enoughMatch between system and the real worldFeedback; Errors; Request for help
    Small screen or font sizeScreen readabilityFeedback; Observations
    Lack of effective color schemeScreen readabilityFeedback; Observations
    Lack of advance options (eg,
    search option)    		Consistency and mappingFeedback; Observations
    No option to (send a message) chat with a clinicianEase of inputFeedback; Observations
    Problem with editing and no obvious go back optionEase of input and Consistency and mappingFeedback; Errors; Request for help; Observations

    Participants through usability testing and the interview commented on the need for a more advanced and attractive design, with better functionality and features in ASyMS to better address the needs of end users, as indicated below:

    Finding where everything is, it’s not labeled, so it would be easier if every option was labeled...I think it needs a higher-level menu, which may have to be categorized, which allows me to navigate around through it easily.
    [Participant 1]
    ...add a search button. Having a search button just kills so much time. You can access the entire database in like 2 seconds. Saves a lot of time...I wish it had a search button.
    [Participant 3]

    Some older participants (age>65) commented that they would prefer to use ASyMS on a device with a larger screen (with larger font size) and a more effective color scheme that better draws users’ attention toward specific elements on the screen.

    A bigger screen for people who need glasses...someone like me whose vision is affected needs a bigger screen
    [Participant 4]
    It is always nice for someone in my age if got a bigger text...Of course, if you could make it bigger, would be great.
    [Participant 7]
    I have an iPhone, which the text isn’t very much bigger than that, right? Like the text is the same basically. But the screen colors, like this screen color to me (shows iPhone) is a lot easier to read than this (shows the ASyMS’s handset).
    [Participant 5]

    As participants were not familiar with the system prior to the usability session and no tutorial of the ASyMS and its functionalities was given, they often felt insecure about their actions and asked for assistance and approval before performing tasks. Most participants mentioned that they would need some time to learn and get familiar with the ASyMS device before they could start to use it regularly. By the end of the usability sessions, when participants had gained some experience with using ASyMS, all participants agreed that with experience in using the ASyMS, they would get familiar with its features and definitely use it more efficiently, as noted in the following participant quote:

    If people use this a few times they will be able to (use and) navigate it easily.
    [Participant 6]

    Usefulness of the Advanced Symptom Management System

    Regarding the self-care advice and information provided in the library, almost every participant commented that the ASyMS provides a lot of quality information. Two participants suggested that using the information would be easier if the self-care information was better categorized.

    The information section, I would rather have that more generalized and categorized. It is easier for the person to use.
    [Participant 6]
    There is no easy way to find information here. You have to read the entire list to get what you want, and probably read more than once. Categorization would improve that.
    [Participant 2]

    One older (aged 70 years) participant, who felt comfortable using a smartphone and had a high level of education suggested that it would be helpful if the information and advice could be customized to the specific therapy that patients receive. In addition, he commented that the self-care information he sought was not covered in the self-care advice as he experienced different treatment toxicities and suggested to modify and enhance the self-care advice.

    It would help if you could customize to the individual...the specific therapy they are receiving...There is one major thing, that is the food. One of the problems in my experience, I went through two different regimens of chemotherapy, different elements of the regimens have different food requirements, some for example do not allow caffeine or alcohol or meat products or spices. None of that is here, that would be helpful if me and my wife are about to contemplate dinner and it tells us can I eat such and such, those answers would be helpful in terms of my chemotherapy. This app tells us about chemotherapy in general, whereas different patients have different regimens of chemotherapy...
    [Participant 8]

    Intention to Use

    All participants mentioned that the ASyMS would be a valuable device to use for managing cancer treatment side effects. Results from the modified TAM and the interviews suggested that almost all participants were satisfied and pleased with their experience in using the ASyMS device, and this positively influenced their attitude toward using the ASyMS in the future. Participants indicated confidence that the problem of communication with their health care provider can be solved (to some extent) by using such a device, and it will help participants to manage their symptoms quicker than the other current available options.

    Clinician is typically in a hurry, not using a lot of words, sometimes very technical words...easy to get snowed...I always try to bring my wife or my son or my daughter, so they can hear what the doctor says and also ask questions. The conversation between me and my doctor is brief and complex. If I have any questions later, I cannot reach them...I call their secretary, who asks that clinician, and gets back to me in a few days...makes it impossible to ask follow-up questions. There is a problem with the nature of this communication, which a system like ASyMS could improve, if it provided for multiple interactions...I can see value in such a system if it provided that capability.
    [Participant 10]
    it actually would give you a pretty good history. And probably you would be able to deal with the nasty symptoms quicker than the other options which is trying to contact your doctor or nurse, and its not that easy.
    [Participant 2]

    Despite the proliferative use of mobile technologies in health care, few Web-enabled PROMs systems have been developed with consideration of their quality through comprehensive and rigorous usability evaluations [38]. While ASyMS has undergone several years of development [17,18], this study has added to the knowledge about usability issues, acceptability, and the potential for the uptake of this mobile technology in Canadian cancer populations to manage the acute effects of cancer treatment.The main general design recommendations (according to usability heuristics) for enhancing features of the ASyMS are as follows: enhance the readability and glanceability of screen; implement advance options (including search option, easy identifiable back option, intuitive pop-up screen option, and advanced navigation options, eg, swiping screens, for expert users); and support navigation by creating an option to customize main menu features, particularly the self-care advice to make it easier to find rather than reading through lengthy text.

    Concerns have been raised in the literature that modern technologies, such as mobile devices, may not be entirely appropriate for use by all cancer populations, as it might be considered difficult to use. For example, older adults may experience difficulties when using technologies such as mobile devices or smartphones [39]. However, there has been a growing interest in the design of technologies, including innovative health technology design for older adults, who often manage complex health conditions and multiple chronic illnesses, to provide better and more sufficient supportive care services [40]. Our study findings demonstrated that older participants (>65) were interested and had a positive attitude toward using the ASyMS device, although a few of them mentioned that they prefer to use the ASyMS on a larger device with larger font size. Furthermore, they mentioned that their performance was affected by age-related physical and mental health status. This is also shown in previous research that older adults are interested and capable of using modern technologies for managing health care issues [41-43].

    Furthermore, limited experience with aspects of mobile technology did not affect the acceptance of the mobile device in this study. This is also consistent with the result of a recent literature review indicating that mobile devices, such as smartphones, can be ideal tools for novice users who have very little understanding of how software or a system in general works, as users learn how to use a touchscreen after a few tries [39]. Although none of the participants had previous experience in using the ASyMS device, all of them became proficient during or by the end of usability testing sessions, indicating that the training period does not need to be long; nevertheless, the incorporation of tutorials and training are important to reduce the time needed by users to learn how to use the system [44,45]. The training should focus on the system features that are more problematic, challenging, and complex for users [43], ensuring that patients feel confident in the use of the system.

    Besides design issues and problems observed in the usability testing, participants also commented on the ASyMS content to enable self-management of treatment symptoms. Previous research has shown that a higher perception of the content richness in a system has resulted in a higher perception of the usefulness of the system [46]. The content richness is defined as the adequacy of resources that users can access to improve their activity on a particular technology [47]. As noted by Lee and Lehto [48], the content richness is a key significant predictor of the perceived usefulness [49]. Our findings support a need to enhance the self-care advice and personalized tailoring to treatment regimens to better support patients in taking the required actions for symptom self-management. Evidence-based guidelines for symptom management have been developed [50] and best practices in presenting information in an “actionable” format should be considered in the future design of the ASyMS device [51]. Furthermore, as patients have different learning styles, the use of an extensive library of written, audio, and video information resources and patient education materials and guidelines for symptom self-management would be beneficial.

    Although the usability considers a combination of factors (including intuitive design, ease of learning, efficiency of use, memorability, error frequency, and user satisfaction [52,53]), one usability evaluation cannot claim to cover all possible and critical usage situations that can possibly occur. Testing the ASyMS in a real-world setting and evaluation of the effectiveness through a trial is needed given high variability in practices [43,54]. We have customized some of the features in the ASyMS device based on the data derived from this study. For instance, we have modified the content of the self-care library to be more action-oriented to foster patient self-management. Currently, a feasibility randomized controlled trial is under way (NCT03335189) that will identify the implementation and context-related issues prior to a larger, multisite randomized controlled trial in Canada.

    Acknowledgments

    Partial funding for the study was supported through the Ontario Patient-Reported Outcomes of Symptoms and Toxicity Research Unit and the Nursing Department at Princess Margaret Cancer Centre. We extend a special thank you and note of appreciation to all patients who participated in this study. We are also grateful to our technology partner, Docobo, for providing us with the Web-based platform and the ASyMS patient handset for usability testing.

    Conflicts of Interest

    None declared.

    ‎

    Multimedia Appendix 1

    ASyMS Monitoring System.

    PDF File (Adobe PDF File), 187KB
    ‎

    Multimedia Appendix 2

    Features of ASyMS.

    PDF File (Adobe PDF File), 682KB
    ‎

    Multimedia Appendix 3

    ASYMS Acceptance Measure.

    PDF File (Adobe PDF File), 42KB
    ‎

    Multimedia Appendix 4

    Usability test script.

    PDF File (Adobe PDF File), 230KB
    1. Davies A, Epstein J. Oral complications of cancer and its management. In: Oral Complications of Cancer and its Management. Great Britain: Oxford University Press, USA; 2010.
    2. Rajapakse D. Symptom management during chemotherapy. Paediatrics and Child Health 2010 Mar 01;20(3):129-134 [FREE Full text] [CrossRef] [Medline]
    3. Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, et al. Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J Clin Oncol 2016 Feb 20;34(6):557-565 [FREE Full text] [CrossRef] [Medline]
    4. Ruland CM, Andersen T, Jeneson A, Moore S, Grimsbø GH, Børøsund E, et al. Effects of an internet support system to assist cancer patients in reducing symptom distress: a randomized controlled trial. Cancer Nurs 2013;36(1):6-17. [CrossRef] [Medline]
    5. Calvin O, Da T. Usability study of a computer-based self-management system for older adults with chronic diseases. JMIR Res Protoc 2012 Nov 08;1(2):e13 [FREE Full text] [CrossRef] [Medline]
    6. Stone AA, Broderick JE. Real-time data collection for pain: appraisal and current status. Pain Med 2007 Oct;8 Suppl 3:S85-S93. [CrossRef] [Medline]
    7. Bender JL, Radhakrishnan A, Diorio C, Englesakis M, Jadad AR. Can pain be managed through the Internet? A systematic review of randomized controlled trials. Pain 2011 Aug;152(8):1740-1750. [CrossRef] [Medline]
    8. Moradian S, Voelker N, Brown C, Liu G, Howell D. Effectiveness of Internet-based interventions in managing chemotherapy-related symptoms in patients with cancer: a systematic literature review. Support Care Cancer 2018 Dec;26(2):361-374. [CrossRef] [Medline]
    9. Tan WS, Liu D, Bishu R. Web evaluation: Heuristic evaluation vs. user testing. International Journal of Industrial Ergonomics 2009 Jul;39(4):621-627. [CrossRef]
    10. Vickers A. Prediction models in cancer care. CA Cancer J Clin 2011;61(5):315-326 [FREE Full text] [CrossRef] [Medline]
    11. Cleeland CS, Wang XS, Shi Q, Mendoza TR, Wright SL, Berry MD, et al. Automated symptom alerts reduce postoperative symptom severity after cancer surgery: a randomized controlled clinical trial. J Clin Oncol 2011 Mar 10;29(8):994-1000 [FREE Full text] [CrossRef] [Medline]
    12. Gilbert JE, Howell D, King S, Sawka C, Hughes E, Angus H, et al. Quality improvement in cancer symptom assessment and control: the Provincial Palliative Care Integration Project (PPCIP). J Pain Symptom Manage 2012 Apr;43(4):663-678. [CrossRef] [Medline]
    13. Bender JL, Yue RYK, To MJ, Deacken L, Jadad AR. A lot of action, but not in the right direction: systematic review and content analysis of smartphone applications for the prevention, detection, and management of cancer. J Med Internet Res 2013 Dec 23;15(12):e287 [FREE Full text] [CrossRef] [Medline]
    14. Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability Testing of a Complex Clinical Decision Support Tool in the Emergency Department: Lessons Learned. JMIR Hum Factors 2015 Sep 10;2(2):e14 [FREE Full text] [CrossRef] [Medline]
    15. Maguire R, Ream E, Richardson A, Connaghan J, Johnston B, Kotronoulas G, et al. Development of a novel remote patient monitoring system: the advanced symptom management system for radiotherapy to improve the symptom experience of patients with lung cancer receiving radiotherapy. Cancer Nurs 2015;38(2):E37-E47. [CrossRef] [Medline]
    16. Maguire R, Fox PA, McCann L, Miaskowski C, Kotronoulas G, Miller M, et al. The eSMART study protocol: a randomised controlled trial to evaluate electronic symptom management using the advanced symptom management system (ASyMS) remote technology for patients with cancer. BMJ Open 2017 Dec 06;7(5):e015016 [FREE Full text] [CrossRef] [Medline]
    17. Kearney N, McCann L, Norrie J, Taylor L, Gray P, McGee-Lennon M, et al. Evaluation of a mobile phone-based, advanced symptom management system (ASyMS) in the management of chemotherapy-related toxicity. Support Care Cancer 2009 Apr;17(4):437-444. [CrossRef] [Medline]
    18. Maguire R, Miller M, Sage M, Norrie J, McCann L, Taylor L, et al. Results of a UK based pilot study of a mobile phone based advanced symptom management system (ASyMS) in the remote monitoring of chemotherapy related toxicity. Clinical Effectiveness in Nursing 2005 Sep;9(3-4):202-210. [CrossRef]
    19. Cowie J, McCann L, Maguire R, Kearney N, Connaghan J, Paterson C, et al. Real-time management of chemotherapy toxicity using the Advanced Symptom Management System (ASyMS). Journal of Decision Systems 2013 Jan;22(1):43-52. [CrossRef]
    20. Green CA, Duan N, Gibbons RD, Hoagwood KE, Palinkas LA, Wisdom JP. Approaches to Mixed Methods Dissemination and Implementation Research: Methods, Strengths, Caveats, and Opportunities. Adm Policy Ment Health 2015 Sep;42(5):508-523 [FREE Full text] [CrossRef] [Medline]
    21. Polit DF. Nursing Research:Generating and Assessing Evidence for Nursing Practice. Philadelphia: Lippincott Williams & Wilkins; 2012.
    22. Nielsen J. Usability Engineering. New York: Academic Press; 1993.
    23. International Organization for Standardization. International Organization for Standardization. Geneva, Switzerland; 2018 Mar. Ergonomics of human-system interaction -- Part 11: Usability: Definitions and concepts   URL: https://www.iso.org/standard/63500.html [accessed 2018-04-29] [WebCite Cache]
    24. Sandelowski M, Barroso J, Voils CI. Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Res Nurs Health 2007 Feb;30(1):99-111 [FREE Full text] [CrossRef] [Medline]
    25. Yen P, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012;19(3):413-422 [FREE Full text] [CrossRef] [Medline]
    26. Hix D, Hartson HR. Developing User Interfaces: Ensuring Usability through Product and Process. New York: John Wiley & Sons; 1993.
    27. Kushniruk AW, Patel VL, Cimino JJ. Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces. Proceedings of the AMIA Annual Fall Symposium 1997:218-222 [FREE Full text] [Medline]
    28. Lewis J. Evaluation of Procedures for Adjusting Problem-Discovery Rates Estimated From Small Samples. International Journal of Human-Computer Interaction 1994;13(4):445-479. [CrossRef]
    29. Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003 Aug;35(3):379-383. [Medline]
    30. Virzi R. Refining the Test Phase of Usability Evaluation:How Many Subjects Is Enough? Human Factors 1992;34(4):457-468. [CrossRef]
    31. Shah SGS, Robinson I. Benefits of and barriers to involving users in medical device technology development and evaluation. Int J Technol Assess Health Care 2007;23(1):131-137. [CrossRef] [Medline]
    32. Yardley L, Morrison L, Andreou P, Joseph J, Little P. Understanding reactions to an internet-delivered health-care intervention: accommodating user preferences for information provision. BMC Med Inform Decis Mak 2010 Sep 17;10:52 [FREE Full text] [CrossRef] [Medline]
    33. Ajzen I. Constructing a Theory of Planned Behaviour questionnaire: Conceptual and methodological considerations. 2006. TPB Questionnaire Construction   URL: http://people.umass.edu/aizen/pdf/tpb.measurement.pdf [accessed 2018-04-29] [WebCite Cache]
    34. Gibbs G. Analysing Qualitative Data. London, England: SAGE PUBN; 2007.
    35. Fereday J, Muir-Cochrane E. Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. International Journal of Qualitative Methods 2016 Nov 29;5(1):80-92. [CrossRef]
    36. O’Reilly M, Parker N. ‘Unsatisfactory Saturation’: a critical exploration of the notion of saturated sample sizes in qualitative research. Qualitative Research 2013;13(2):190-197. [CrossRef]
    37. Bertini E, Gabrielli S, Kimani S, Catarci T, Santucci G. Appropriating and assessing heuristics for mobile computing. 2006 Presented at: the working conference on Advanced visual interfaces. Venezia, Italy: ACM; 2006; Venezia, Italy p. 119-126. [CrossRef]
    38. Cho H, Powell D, Pichon A, Thai J, Bruce J, Kuhns LM, et al. A Mobile Health Intervention for HIV Prevention Among Racially and Ethnically Diverse Young Men: Usability Evaluation. JMIR Mhealth Uhealth 2018 Sep 07;6(9):e11450 [FREE Full text] [CrossRef] [Medline]
    39. Zapata B, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst 2015 Feb;39(2):1. [CrossRef] [Medline]
    40. Sakaguchi-Tang DK, Bosold AL, Choi YK, Turner AM. Patient Portal Use and Experience Among Older Adults: Systematic Review. JMIR Med Inform 2017 Oct 16;5(4):e38 [FREE Full text] [CrossRef] [Medline]
    41. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res 2012 Jan 27;14(1):e19 [FREE Full text] [CrossRef] [Medline]
    42. Yang HH, Yu C, Huang CH, Kuo LH, HJ. Teaching information technology for elder participation: A qualitative analysis of Taiwan retirees. WSEAS Transactions on Information Science and Applications 2010 Sep;7(9):1190-1199 [FREE Full text]
    43. Mirkovic J, Kaufman DR, Ruland CM. Supporting cancer patients in illness management: usability evaluation of a mobile app. JMIR Mhealth Uhealth 2014 Aug 13;2(3):e33 [FREE Full text] [CrossRef] [Medline]
    44. Zargaran E, Schuurman N, Nicol AJ, Matzopoulos R, Cinnamon J, Taulu T, et al. The electronic Trauma Health Record: design and usability of a novel tablet-based tool for trauma care and injury surveillance in low resource settings. J Am Coll Surg 2014 Jan;218(1):41-50. [CrossRef] [Medline]
    45. Hwang Y, Shin D, Yang CY, Lee SY, Kim J, Kong B, et al. Developing a Voice User Interface with Improved Usability for People with Dysarthria. In: Miesenberger K, Karshmer A, Penaz P, Zagler W, editors. Computers Helping People with Special Needs:13th International Conference, ICCHP 2012, Linz, Austria. Berlin, Heidelberg: Springer; 2012:117-124.
    46. Chen Y, Lan Y. An Empirical Study of the Factors Affecting Mobile Shopping in Taiwan. Int. J. Technol. Hum. Interact 2014;10:19-30. [CrossRef]
    47. Bargshady G, Pourmahdi K, Khodakarami P, Khodadadi T, Alipanah F. The effective factors on user acceptance in mobile business intelligence. Jurnal Teknologi (Sciences & Engineering) 2015;72:49-54. [CrossRef]
    48. Dy L, Lehto M. User acceptance of YouTube for procedural learning: An extension of the Technology Acceptance Model. Computers & Education 2013 Feb;61:193-208. [CrossRef]
    49. Pindeh N, Suki N, Suki N. User Acceptance on Mobile Apps as an Effective Medium to Learn Kadazandusun Language. Procedia Economics and Finance 2016;37:372-378. [CrossRef]
    50. Cancer Care Ontario. https://www.cancercareontario.ca/en/guidelines-advice. 2018. Guidelines & Advice   URL: https://www.cancercareontario.ca/en/guidelines-advice [accessed 2018-04-29] [WebCite Cache]
    51. Barr J, Fraser GL, Puntillo K, Ely EW, Gélinas C, Dasta JF, American College of Critical Care Medicine. Clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the intensive care unit. Crit Care Med 2013 Jan;41(1):263-306. [CrossRef] [Medline]
    52. Aniobi DE, Alu ES. Usability Testing and Evaluation of a Cloud Computing-based Mobile Learning Apptudents' Perspective. International Journal of Computer Science Issues (IJCSI) 2016;13:67.
    53. Azhar FAB, Dhillon JS. A systematic review of factors influencing the effective use of mHealth apps for self-care. 2016 Presented at: 3rd International Conference on Computer and Information Sciences (ICCOINS); 2016; Kuala Lumpur p. 191-196. [CrossRef]
    54. Rogers Y, Connelly K, Tedesco L, Hazlewood W, Kurtz A, Hall RE, et al. Why It’s Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp. In: Krumm J, Abowd G, Seneviratne A, Strang T, editors. UbiComp 2007: Ubiquitous Computing: 9th International Conference, UbiComp 2007, Innsbruck, Austria, September 16-19, 2007, Proceedings. Berlin, Heidelberg: Springer; 2007:353.

    ‎
    ASyMS: Advanced Symptom Management System
    PROM: patient-reported outcome measure
    TAM: Telehealth Acceptance Measure
    UHN: University Health Network

    Edited by G Eysenbach; submitted 03.05.18; peer-reviewed by G Matthias, SW Bae, P Chow; comments to author 17.08.18; revised version received 05.10.18; accepted 30.10.18; published 21.12.18

    Copyright

    ©Saeed Moradian, Monika K. Krzyzanowska, Roma Maguire, Plinio P. Morita, Vishal Kukreti, Jonathan Avery, Geoffrey Liu, Joseph Cafazzo, Doris Howell. Originally published in JMIR Cancer (http://cancer.jmir.org), 21.12.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Cancer, is properly cited. The complete bibliographic information, a link to the original publication on http://cancer.jmir.org/, as well as this copyright and license information must be included.